1. Field of the Invention
The present invention relates to a neural network, and more particularly to a neural network that can be trained to learn a linearly unseparable input pattern and output values therefor (the combination of such a linearly unseparable input pattern and output values therefor will be referred to as an "input/output pattern"), and a learning method for such a neural network.
2. Description of the Background Art
Various models for neural networks have heretofore been proposed. It is known that the perceptron is the most basic neural network model. The perceptron is based on the learning process in which only the weight of association (association coefficient) between input and output layers is varied. The learning process of the perceptron is advantageous in that it is simpler than other learning processes. The learning process is a process in which the weight is varied to get the output to approach a target output.
However, the perceptron has a problem in that it cannot be trained to learn linearly unseparable patterns. If a pattern that can be classified into two kinds is linearly separable, it can be separated into the classifications in a straight line when it is two-dimensional and in a multidimensional plane when it is three-dimensional or more.
The problem of the perceptron is that linearly unseparable patterns that cannot be learned can be solved by a process known as back propagation. The back propagation is applicable to a multilayer neural network model, and allows linearly unseparable patterns to be learned. The back propagation basically serves to minimize the square function of an error between a target output and an actual output, and uses an optimization process referred to as the method of steepest descent. Therefore, if there is a local minimum present in an error function obtained from given input/output patterns, then a learning failure may result when trapped in the local minimum. The back propagation is also disadvantageous in that it is cumbersome to adjust parameters and initialize the weight, and difficult to determine the number of necessary neurons of a hidden layer between input and output layers, and the process requires a large amount of calculation and is time-consuming.
There have also been proposed algorithms for adding neuron units needed to construct a multilayer neural network.
One of the proposed algorithms serves to determine the number of neuron unit layers of a feedforward hierarchical network and the number of neuron units in each of the neuron unit layers. It adds neuron unit layers or neuron units in neuron unit layers until convergence is reached. For details, see The Upstart Algorithm: A method for Constructing and Training Feedforward Neural Networks, written by Marcus Frean, Neural Computation, Vol. 2, pages 198-209, 1990, Massachusetts Institute of Technology.
According to another proposed algorithm, neuron units are added as required according to predetermined rules in order to build a multilayer perceptron composed of linear threshold units. For details, see Learning in feed-forward layered networks: the tiling algorithm, written by Marc Mezard and Jean-Pierre Nadal, Journal of Physics A: Math. Gen. Vol. 22, pages 2191-2203, 1989.
The above proposed algorithms add necessary neuron units until a desired output is obtained with respect to a given input. Therefore, neuron layers and neurons thereof which are required are not determined until a neural network is finally constructed, and the number of neuron layers used and the number of neurons used tend to be large. As a result, the resultant neuron networks are liable to be complex in structure, and do not lend themselves to high-speed processing operation.